------------------------------ HOME EQUITY LOAN DATASET -----------------------------------¶
Context:¶
The consumer credit department of a bank wants to automate the decisionmaking process for approval of home equity lines of credit. To do this, they will follow the recommendations of the Equal Credit Opportunity Act to create an empirically derived and statistically sound credit scoring model. The model will be based on data collected from recent applicants granted credit through the current process of loan underwriting. The model will be built from predictive modeling tools, but the created model must be sufficiently interpretable to provide a reason for any adverse actions (rejections).
Content:¶
The Home Equity dataset (HMEQ) contains baseline and loan performance information for 5,960 recent home equity loans. The target (BAD) is a binary variable indicating whether an applicant eventually defaulted or was seriously delinquent. This adverse outcome occurred in 1,189 cases (20%). For each applicant, 12 input variables were recorded.
In [1]:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.express as px
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
data = pd.read_csv('hmeq.csv')
In [5]:
688/891
Out[5]:
In [3]:
data.shape
Out[3]:
In [2]:
data.isnull().sum()
Out[2]:
1. [BASIC INFORMATION]¶
In [2]:
data.head()
Out[2]:
- BAD (Bi): 1 = loan defaulted, 0 = loan repaid
- LOAN (Cont): amount of the loan request
- MORTDUE (Cont): amount due on existing mortgage
- VALUE (Cont): value of current property
- REASON (Cat): DebtCon = debt consolidation, HomeImp = home improvement
- JOB (Cat): six occupational categories
- YOJ (Cont): years at present job
- DEROG (Cont): number of major derogatory reports
- DELINQ (Cont): number of delinquent credit lines
- CLAGE (Cont): age of oldest trade line in months
- NINQ (Cont): number of recent credit lines
- CLNO (Cont): number of credit lines
- DEBTINC (Cont): Debt-to-income ratio
2. [NULL VALUES ANALYSIS]¶
In [3]:
# data overview (especially on nan & nan percentage)
listItem = []
for col in data.columns :
listItem.append([col, data[col].dtype, data[col].isna().sum(), round((data[col].isna().sum()/len(data[col])) * 100,2),
data[col].nunique(), list(data[col].drop_duplicates().sample(2).values)]);
dfDesc = pd.DataFrame(columns=['Data Features', 'Data Type', 'Null Count', 'Null %', 'N-Unique', 'Unique Sample'],
data=listItem)
dfDesc
Out[3]:
In [4]:
## rows with nulls
len(data[data.isnull().any(axis=1)])
Out[4]:
In [5]:
## rows without nulls
len(data[~data.isnull().any(axis=1)])
Out[5]:
In [6]:
## the dataset with nulls
data[data.isnull().any(axis=1)]
Out[6]:
In [4]:
## correlate null features
coba = data.copy()
coba['MORTDUE NULL'] = coba['MORTDUE'].isna()
coba['VALUE NULL'] = coba['VALUE'].isna()
coba['REASON NULL'] = coba['REASON'].isna()
coba['JOB NULL'] = coba['JOB'].isna()
coba['YOJ NULL'] = coba['YOJ'].isna()
coba['DEROG NULL'] = coba['DEROG'].isna()
coba['DELINQ NULL'] = coba['DELINQ'].isna()
coba['CLAGE NULL'] = coba['CLAGE'].isna()
coba['CLNO NULL'] = coba['CLNO'].isna()
coba['DEBTINC NULL'] = coba['DEBTINC'].isna()
plt.figure(figsize= (20,20))
sns.heatmap(coba.corr('pearson'), annot= True)
Out[4]:
In [7]:
# trying to find the null combinations
pd.options.display.max_rows = None
null = []
for idx in range(len(data)):
temp = []
for i in data.columns:
if (str(data[i].iloc[idx]).lower() == 'nan'):
temp.append(i)
else:
pass
temp.sort()
null.append(temp)
print(len(null))
(pd.Series(null)).value_counts()
Out[7]:
From the observations:
- It turns out that DEBTINC & DEROG columns has more than 10% nan values, where DEBTINC columns even has at least 20% nan value
- DEROG columns' information may be similar in essence with DELINQ columns' information, thereby safe to be dropped (since it also has a lot of nan values)
- The nan rows & columns appears to be at Random, thereby creating noise. To avoid bias (by imputing nan values), we could drop the rest of nan values
In [2]:
# creating data contingency
mydata = data.copy(deep=True)
# drop debtinc & derog columns because > 10% nan
mydata = mydata.drop(['DEBTINC','DEROG'], axis = 1)
# drop nan rows to eliminate noise
mydata = mydata.dropna()
# resetting index after dropna
mydata = mydata.reset_index(drop=True)
# final data shape
mydata.shape
Out[2]:
In [8]:
# saving mydata (clean data) to csv for further usage
mydata.to_csv('mydata.csv', index= False)
In [9]:
# final nan values checking
mydata.isnull().sum()
Out[9]:
3. [EXPLORATORY DATA ANALYSIS]¶
In [88]:
# creating a pairplot with my target variable as the hue
plt.figure(figsize=(40,30))
sns.pairplot(mydata, hue = 'BAD', diag_kind = 'hist')
Out[88]:
In [11]:
# groupby BAD description for numerical features
mydata.groupby(['BAD']).describe().T
Out[11]:
In [14]:
# whole data description for numerical features
mydata.describe().T
Out[14]:
From the observations:
- It is quite interesting to note that, in all of the features, the group means for 0 category (loan repaid) is higher than the 1 category (loan defaulted) except for the DELINQ & NINQ features.
- DELINQ stands for the amount of delinquent credit lines. In this groupby analysis, we can see that the 0 & 1 category has a group mean of 0,26 & 1,20 respectively. Furthermore, the mean of 1,20 is even higher than the mean of the whole which is 0,44. It means that the 1 category is more likely to have more delinquent credit lines.
- NINQ stands for the amount of recent credit lines. In this groupby analysis, we can see that the 0 & 1 category has a group mean of 1,04 & 1,85 respectively. Furthermore, the mean of 1,85 is even higher than the mean of the whole which is 1,20. It means that the 1 category is more likely to have a slightly more number of recent credit lines.
- It is interesting what these two features may bring, so let's keep these two in mind for the next steps.
In [11]:
# groupby description for categorical values
mydata.groupby(['BAD']).describe(exclude = 'number').T
Out[11]:
In [15]:
# whole data description for categorical values
mydata.describe(exclude='number').T
Out[15]:
- It could be said that there's a significant amount that shows the reason for equity loan (REASON) is "Debt Consolidation" & their job category (JOB) is "Other".
- I'm interested with this many JOB categories, let's find out more about it.
In [91]:
# grouped by JOB description
pd.options.display.max_rows = 72
mydata.groupby(['JOB']).describe().T
Out[91]:
In [96]:
# on JOB-BAD
sns.set_style("whitegrid")
plt.figure(figsize= (18,10))
plt.subplot(2,3,1)
sns.countplot(data = mydata, x='JOB')
plt.subplot(2,3,2)
sns.countplot(data = mydata, x='JOB' , hue = 'BAD')
plt.subplot(2,3,3)
sns.barplot(data = mydata, x='JOB', y='BAD')
Out[96]:
- It appears that "Other" is the highest in terms of frequency. Accordingly, the amount of loan default (BAD = 1) reaches the highest frequency in this "Other" category. Nevertheless, the highest group mean towards loan default is the "Self" category (mean = 0.37). Perhaps, it is due to its sample size which is the smallest amongst all that could help it in reaching the highest means.
In [82]:
# on JOB-LOAN-BAD
plt.figure(figsize= (13,10))
plt.subplot(2,2,1)
sns.barplot(data = mydata, x='JOB', y= 'LOAN')
plt.subplot(2,2,2)
sns.barplot(data = mydata, x='JOB', y= 'LOAN' , hue = 'BAD')
Out[82]:
- In terms of LOAN, it appears that the "Self" category takes the first prize. It reaches the mean value of USD31.264, perhaps, due to its sample size which is the second smallest. If we take a look in terms of LOAN max value, it is taken by the "Other" category with the LOAN value of USD89.900. But then it's interesting to note that SELF, with its highest LOAN mean, has also the highest LOAN mean that defaults.
In [79]:
# on JOB-DELINQ-BAD
plt.figure(figsize= (13,10))
plt.subplot(2,2,1)
sns.barplot(data = mydata, x='JOB', y= 'DELINQ' )
plt.subplot(2,2,2)
sns.barplot(data = mydata, x='JOB', y= 'DELINQ' , hue = 'BAD')
Out[79]:
- In terms of DELINQ, it appears that the "Self" category takes the first prize again. It reaches the DELINQ mean value of 0.589 credit lines. Moreover, it also has the highest DELINQ max value, with the value of 15 credit lines. It's interesting to see that this "Self" category has the lowest DELINQ mean that repays, and at the same time, has the highest DELINQ mean that defaults
In [80]:
# on JOB-NINQ-BAD
plt.figure(figsize= (13,10))
plt.subplot(2,2,1)
sns.barplot(data = mydata, x='JOB', y= 'NINQ')
plt.subplot(2,2,2)
sns.barplot(data = mydata, x='JOB', y= 'NINQ' , hue = 'BAD')
Out[80]:
- In terms of NINQ, it appears that the "Mgr" category takes the first prize. It reaches the NINQ mean value of 1.55 recent credit lines. Special note on the "Self" Category that reachest the second highest NINQ mean of 1.48 recent credit lines. Nevertheless, the highest NINQ max value is taken by the "Office" Category with the value of 13 recent credit lines. In terms of loan defaults, the NINQ mean of each JOB categories does not differ a lot. On the other hand, for loan repayments, the NINQ mean of "Sales" category seems very distinctly the lowest.
In [83]:
# on JOB-MORTDUE-BAD
plt.figure(figsize= (13,10))
plt.subplot(2,2,1)
sns.barplot(data = mydata, x='JOB', y= 'MORTDUE')
plt.subplot(2,2,2)
sns.barplot(data = mydata, x='JOB', y= 'MORTDUE' , hue = 'BAD')
Out[83]:
- In terms of MORTDUE, it appears that the "Self" category, again, takes the first prize. It reaches the MORTDUE mean value of USD111.610. Nevertheless, the highest MORTDUE max value is taken by the "ProfExe" category with the value of USD399.412. Moreover, in terms of loan defaults, the MORTDUE mean of "ProfExe" category is also the highest one. Special note on the MORTDUE mean in "Self" category that repays for it reaches the highest MORTDUE means.
In [84]:
# on JOB-CLAGE-BAD
plt.figure(figsize= (13,10))
plt.subplot(2,2,1)
sns.barplot(data = mydata, x='JOB', y= 'CLAGE')
plt.subplot(2,2,2)
sns.barplot(data = mydata, x='JOB', y= 'CLAGE' , hue = 'BAD')
Out[84]:
- In terms of CLAGE, it appears that the "ProfExe" & "Self" category takes the first prize & second prize with close differences. It reaches the CLAGE mean value of 197.6 months & 197.2 months, respectively. Nevertheless, the highest MORTDUE max value is taken by the "Other" category with the value of 1168 months. In terms of loan defaults, the CLAGE mean of JOB categories appears to not differ a lot. Special note on the CLAGE mean in "Self" category that repays for it reaches the highest CLAGE means.
3.1 [CRAMER'S V ]¶
In [16]:
# defining cramers v function to see the association between two categorical features
def cramers_v(x,y):
import scipy.stats as ss
confusion_matrix = pd.crosstab(x,y)
chi2 = ss.chi2_contingency(confusion_matrix)[0]
n = confusion_matrix.sum().sum()
phi2 = chi2/n
r,k = confusion_matrix.shape
phi2corr = max(0, phi2-((k-1)*(r-1))/(n-1))
rcorr = r -((r-1)**2)/(n-1)
kcorr = k -((k-1)**2)/(n-1)
return np.sqrt(phi2corr/min((kcorr-1),(rcorr-1)))
In [16]:
# cramers v value between REASON & BAD
cramers_v(mydata['REASON'], mydata['BAD'])
Out[16]:
In [17]:
# cramers v value between JOB & BAD
cramers_v(mydata['JOB'], mydata['BAD'])
Out[17]:
In terms of Cramer's V:
- There seems to be no or very weak association between REASON (reason for loan) & BAD (repaid or default)
- There seems to be a moderate association between JOB (job category) & BAD
3.2 [CORRELATION RATIO]¶
In [25]:
# defining correlation ratio function to see the association between numerical-categorical features
def correlation_ratio(categories, measurements):
fcat, _ = pd.factorize(categories)
cat_num = np.max(fcat) + 1
y_avg_array = np.zeros(cat_num)
n_array = np.zeros(cat_num)
for i in range(0, cat_num):
cat_measures = measurements[np.argwhere(fcat==i).flatten()]
n_array[i] = len(cat_measures)
y_avg_array[i] = np.average(cat_measures)
y_total_avg = np.sum(np.multiply(y_avg_array, n_array))/np.sum(n_array)
numerator = np.sum(np.multiply(n_array, np.power(np.subtract(y_avg_array, y_total_avg),2)))
denominator = np.sum(np.power(np.subtract(measurements, y_total_avg),2))
if numerator == 0:
eta = 0.0
else:
eta = np.sqrt(numerator/denominator)
return eta
from scipy.stats import shapiro, anderson, skew, kurtosis
for i in mydata.drop('BAD',axis = 1).select_dtypes(exclude = 'object').columns:
print(i)
print('Correlation Ratio: {}'.format(correlation_ratio(mydata['BAD'], mydata[i])))
if shapiro(mydata[i])[1] < 0.05:
print('Non-Normal Distribution')
else:
print('Normal Distribution')
print('Skewness: {}, Kurtosis {}'.format(skew(mydata[i]), kurtosis(mydata[i])))
print('\n')
In terms of Correlation Ratio:
- There seems to be a moderate association between DELINQ (amount of delinquent credit lines) & BAD (repaid or default)
- There seems to be a moderate association between NINQ (amount of recent credit lines) & BAD
- There seems to be a moderate association between CLAGE (age of oldest credit line in months) & BAD
- There seems to be a weak or no association between the rest of the features & BAD
In terms of Shapiro value:
- All the numerical features has a non-normal distribution
3.3 [PEARSON CORRELATION]¶
In [19]:
# plotting pearson correlation heatmap
plt.figure(figsize=(8,10))
sns.heatmap(mydata.corr(),annot =True)
Out[19]:
In terms of Pearson Correlation:
- There seems to be a moderate (positive) linear relationship between DELINQ (amount of delinquent credit lines) & BAD (repaid or default)
- There seems to be a weak (positive) linear relationship between NINQ (amount of recent credit lines) & BAD
- There seems to be a weak (negative) linear relationship between CLAGE (age of oldest credit line in months) & BAD
- There seems to be a very weak or no linear relationship between the rest of the features & BAD
3.4 [SPEARMAN CORRELATION]¶
In [20]:
# plotting spearman correlation heatmap
plt.figure(figsize=(8,10))
sns.heatmap(mydata.corr(method = 'spearman'),annot =True)
Out[20]:
In terms of Spearman Correlation:
- There seems to be a moderate (positive) monotonic relationship between DELINQ (amount of delinquent credit lines) & BAD (repaid or default)
- There seems to be a weak (positive) monotonic relationship between NINQ (amount of recent credit lines) & BAD
- There seems to be a weak (negative) monotonic relationship between CLAGE (age of oldest credit line in months) & BAD
- There seems to be a weak (negative) monotonic relationship between LOAN (amount of loan requested) & BAD
- There seems to be a weak (negative) monotonic relationship between VALUE (value of the existing property) & BAD
- There seems to be a very weak or no monotonic relationship between the rest of the features & BAD
3.5 [HYPOTHESIS TESTING]¶
In [18]:
## A/B testing for categorical-categorical columns
from scipy.stats import chi2_contingency
chi2_check = []
categorical_columns = mydata.select_dtypes('object').columns
for i in categorical_columns:
if chi2_contingency(pd.crosstab(mydata['BAD'], mydata[i]))[1] < 0.05:
chi2_check.append('Accept H1')
else:
chi2_check.append('Accept H0')
chi = pd.DataFrame(data = [categorical_columns, chi2_check]).T
chi.columns = ['Column', 'Hypothesis']
chi
Out[18]:
In terms of Chi-Square test:
- since H1 is accepted, there seems to be an influencial relationship between JOB (job category) & BAD (repaid or default)
- since H0 is accepted, there seems to be no influencial relationship between REASON (loan reason) & BAD
In [19]:
## A/B testing for categories within categorical columns
check = {}
for i in chi[chi['Hypothesis'] == 'Accept H1']['Column']:
dummies = pd.get_dummies(mydata[i])
bon_p_value = 0.05/mydata[i].nunique()
for series in dummies:
if chi2_contingency(pd.crosstab(mydata['BAD'], dummies[series]))[1] < bon_p_value:
check['{}_{}'.format(i, series)] = 'Accept H1'
else:
check['{}_{}'.format(i, series)] = 'Accept H0'
res_chi = pd.DataFrame(data = [check.keys(), check.values()]).T
res_chi.columns = ['Pair', 'Hypothesis']
res_chi
Out[19]:
In terms of Chi-Square test:
- Since H0 is accepted, there seems to be no relationship between JOB_Mgr (managerial jobs) & BAD
- Since H1 is accepted, there seems to be a relationship between the rest of JOB categories & BAD
In [20]:
# picking only the ones that Accepts H1
res_chi[res_chi['Hypothesis'] == 'Accept H1']
Out[20]:
In [21]:
# plotting only the ones that Accepts H1
for i in res_chi[res_chi['Hypothesis'] == 'Accept H1']['Pair']:
sns.countplot(mydata[mydata[i.split('_')[0]] == (i.split('_')[1])]['BAD'])
plt.title(i)
plt.show()
In [23]:
## A/B testing for categorical-continuous columns
from scipy.stats import mannwhitneyu
mann = []
for i in mydata.drop('BAD', axis=1).select_dtypes('number').columns:
if mannwhitneyu(mydata[mydata['BAD'] == 0][i],
mydata[mydata['BAD'] == 1][i])[1] < 0.05:
mann.append('Accept H1')
else:
mann.append('Accept H0')
res = pd.DataFrame(data = [list(mydata.drop('BAD', axis=1).select_dtypes('number').columns), mann]).T
res.columns = ['Columns', 'Hypothesis']
res
Out[23]:
In terms of Mann-Whitney U test:
- Since H0 is accepted, there seems to be no significant difference in data distribution within BAD categories in terms of CLNO value. It could also be said that there's an equal probability that a randomly drawn observation from one BAD category is larger than a randomly drawn observation from the other category.
- Since H1 is accepted, there seems to be significant differences in data distribution within BAD categories in terms of LOAN, MORTDUE, VALUE, YOJ, DELINQ, CLAGE, and NINQ value. It could also be said that there's an unequal probability that a randomly drawn observation from one BAD category is larger than a randomly drawn observation from the other category.
In [24]:
# picking only the ones that Accepts H1
res[res['Hypothesis'] == 'Accept H1']
Out[24]:
4. [MODEL DEVELOPMENT]¶
In [3]:
# creating dummy variables so all the data are numerical
mydummy = pd.get_dummies(data= mydata, drop_first= True, columns = ['REASON','JOB'])
mydummy.columns
Out[3]:
In [4]:
# selecting features
IV = ['LOAN', 'MORTDUE', 'VALUE', 'YOJ', 'DELINQ', 'CLAGE', 'NINQ',
'CLNO', 'REASON_HomeImp', 'JOB_Office', 'JOB_Other', 'JOB_ProfExe',
'JOB_Sales', 'JOB_Self']
# independent variables
x = mydummy[IV]
# dependent/target variable
y = mydummy['BAD']
4.1 [OUTLIER-CLUSTERING ANALYSIS]¶
In [42]:
# first, lets use the boxplot to show how the data is distributed
plt.figure(figsize = (8,8))
sns.boxplot(data = mydummy)
plt.xticks(rotation = 90)
Out[42]:
- From the boxplot we could see that the scale of the data isn't similar. Let's rescale the data by using the StandardScaler and check it one more time
In [43]:
# creating dummy copy
from sklearn.preprocessing import StandardScaler
dummy = mydummy.copy()
# rescaling the data
scaler = StandardScaler()
dummy = scaler.fit_transform(dummy)
dummy = pd.DataFrame(dummy, columns = mydummy.columns)
dummy = dummy.drop('BAD', axis=1)
dummy.head()
Out[43]:
In [44]:
# after rescaling, lets use the boxplot to show how the data is now distributed
plt.figure(figsize = (8,8))
sns.boxplot(data = dummy)
plt.xticks(rotation = 90)
Out[44]:
- From the boxplot we could see that the scale of the data is now, more or less, on a level.
In [45]:
# let'use the PCA to reduce our 14 features to 10 columns
from sklearn.decomposition import PCA
pca = PCA(n_components=10, random_state=101)
pca.fit(dummy)
x_pca = pca.transform(dummy)
In [46]:
# displaying PCA columns
x_pca = pd.DataFrame(x_pca, columns = ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10'])
x_pca
Out[46]:
In [47]:
# describing the explanation ratio of each PC line
pca.explained_variance_ratio_
Out[47]:
From the pca explained variance ratio function, we could see that:
- PC1 could explain 19.00 % of the data
- PC2 could explain 10.23 % of the data
- PC3 could explain 9.34 % of the data
- PC4 could explain 9.00 % of the data
- PC5 could explain 8.25 % of the data
- PC6 could explain 7.77 % of the data
- PC7 could explain 7.43 % of the data
- PC8 could explain 6.30 % of the data
- PC9 could explain 6.12 % of the data
- PC10 could explain 5.71 % of the data
In [48]:
# sum the total pca explained variance ratio (in %)
sum(pca.explained_variance_ratio_)
Out[48]:
- All the PC line in our analysis could explain in total 89.15 % of our data
In [49]:
# inserting the BAD column into the PCA columns
x_pca['BAD'] = mydummy['BAD']
In [79]:
# inserting a new column based on HDBSCAN labels
from hdbscan import HDBSCAN
hdbscan = HDBSCAN(min_cluster_size = 2, min_samples = 25)
hdbscan.fit(x_pca[['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10']])
x_pca['HDBScan'] = hdbscan.labels_
In [86]:
# identifying the numbers of clusters
n_clusters = len(set(hdbscan.labels_)) - (1 if -1 in hdbscan.labels_ else 0)
n_clusters
Out[86]:
In [87]:
# identifying the numbers of noise/outliers
n_noise = list(hdbscan.labels_).count(-1)
n_noise
Out[87]:
In [88]:
# plotting the minimun spanning tree
hdbscan = HDBSCAN(min_cluster_size=3, gen_min_span_tree=True)
hdbscan.fit(x_pca[['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10']])
plt.figure(figsize=(12,8))
hdbscan.minimum_spanning_tree_.plot(edge_cmap='viridis',
edge_alpha=0.6,
node_size=80,
edge_linewidth=2)
Out[88]:
In [82]:
# plotting the clusters
f, (ax1, ax2) = plt.subplots(1,2, sharey = True, figsize = (15,6))
ax1.set_title('Original')
ax1.scatter(x_pca['PC1'], x_pca['PC2'], x_pca['PC3'])
ax2.set_title('HDBSCAN')
ax2.scatter(x_pca['PC1'], x_pca['PC2'], x_pca['PC3'],c=x_pca['HDBScan'], cmap = 'rainbow')
Out[82]:
In [83]:
# pairplotting the whole data
df = mydata.copy()
df['HDBSCAN LABEL'] = x_pca['HDBScan']
sns.pairplot(df, hue = 'BAD', diag_kind = 'hist')
Out[83]:
In [84]:
# showing the grouped by clusters description (outlier = -1)
pd.options.display.max_rows = 121
df.groupby('HDBSCAN LABEL').describe().T
Out[84]:
In [59]:
df.groupby('HDBSCAN LABEL').describe(exclude='number').T
Out[59]:
From the observations:
- HDBSCAN has identified a cluster of outliers identified with the label of -1 & amounts to 37 datapoints
- This cluster is interesting because they have the highest group mean that approximates the loan default value of 1 (BAD category), with the mean value of 0.5, which also means that this cluster has the highest tendency to default, in relative to other clusters.
- If we dig a little deeper, the outlier cluster starts to show intriguing numbers such as:
- highest loan requested amount (LOAN) mean of USD 45.143 - highest amount due on existing mortgage (MORTDUE) mean of USD 224.648 - highest current property value (VALUE) mean of USD 307.784 - highest amount of delinquent credit lines (DELINQ) mean of 2,5 credit lines - highest age of credit/trade line in months (CLAGE) mean of 283,6 months - highest amount of recent credit lines (NINQ) mean of 2,7 credit lines - highest JOB category of "Self" - An interesting note is that DELINQ, as we've reviewed & analyzed before, has the highest support for its connection towards loan default. If this cluster has the highest mean of DELINQ, we should take precaution. Not to mention that they have higher amount of loan requested, of existing mortgage, of credit line months, and of recent credit lines, which probably makes them tougher on their credit scorings.
- In other words, if a member of this cluster applies for a loan, they should be considered with greater caution for they have a bigger likelihood to default
- For further predictive analysis, let's start to make our prediction models & split our data
4.2 [SPLITTING DATA]¶
In [5]:
# splitting the training data - test data = 80% : 20%
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state= 101)
4.3 [SMOTE RESAMPLING]¶
In [6]:
# create a new x_train & y_train variable (x_trainres & y_trainres) that is resampled using SMOTE method
from imblearn.over_sampling import SMOTE
from collections import Counter
y_train = y_train.astype('int')
smo = SMOTE(random_state=0, sampling_strategy='minority')
x_trainres, y_trainres = smo.fit_resample(x_train, y_train)
print(sorted(Counter(y_trainres).items()))
In [10]:
# independent feature train shape
x_train.shape
Out[10]:
In [11]:
# independent feature resampled train shape
x_trainres.shape
Out[11]:
4.4 [DECISION TREE REPORTS]¶
In [11]:
# model fitting for normal data
from sklearn.tree import DecisionTreeClassifier
dtree = DecisionTreeClassifier()
dtree.fit(x_train,y_train)
Out[11]:
In [12]:
# model fitting for oversampled data
dtree2 = DecisionTreeClassifier()
dtree2.fit(x_trainres,y_trainres)
Out[12]:
In [14]:
# classification report for normal data
from sklearn.metrics import classification_report,confusion_matrix
dtree_pred = dtree.predict(x_test)
dtree_predprob = dtree.predict_proba(x_test)
print(classification_report(y_test, dtree_pred))
# classification report for oversampled data
dtree_pred2 = dtree2.predict(x_test)
dtree_predprob2 = dtree2.predict_proba(x_test)
print(classification_report(y_test, dtree_pred2))
In [58]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, dtree_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, dtree_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [37]:
from sklearn import metrics
# ROC - AUC Score for normal data
dtree_pred_proba = dtree.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, dtree_pred_proba)
auc = metrics.roc_auc_score(y_test, dtree_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
dtree_pred_proba2 = dtree2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, dtree_pred_proba2)
auc = metrics.roc_auc_score(y_test, dtree_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [42]:
from sklearn.model_selection import cross_val_score
# Cross Validation score for normal data
dtreescores = cross_val_score(estimator=dtree,
X=x_train,
y=y_train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(dtreescores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(dtreescores)), dtreescores)
plt.axhline(y=np.mean(dtreescores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
dtreescores2 = cross_val_score(estimator=dtree2,
X=x_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(dtreescores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(dtreescores2)), dtreescores2)
plt.axhline(y=np.mean(dtreescores2), color='g') # Mean value of cross validation scores
plt.show()
In [38]:
from IPython.display import Image
from sklearn.externals.six import StringIO
from sklearn.tree import export_graphviz
import pydot
dot_data = StringIO()
export_graphviz(dtree, out_file = dot_data, feature_names = x_train.columns, filled=True, rounded=True, special_characters=True)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
Image(graph[0].create_png())
Out[38]:
- We can see that the root node is taken by the DELINQ feature. It shows that the DELINQ is the main feature being used in the gaining information thorugh splitting of the nodes in this normal data.
In [39]:
dot_data = StringIO()
export_graphviz(dtree2, out_file = dot_data, feature_names = x_trainres.columns, filled=True, rounded=True, special_characters=True)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
Image(graph[0].create_png())
Out[39]:
- We can see that the root node is once again taken by the DELINQ feature. It shows that the DELINQ is still the main feature being used in gaining more information through the splitting of the nodes in this oversampled data
In [40]:
from eli5 import show_weights
from eli5.sklearn import PermutationImportance
# permutation importance for normal data
dtreeperm = PermutationImportance(dtree, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(dtreeperm, feature_names = list(x_test.columns))
Out[40]:
In [41]:
# permutation importance for oversampled data
dtreeperm2 = PermutationImportance(dtree2, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(dtreeperm2, feature_names = list(x_test.columns))
Out[41]:
- From the permutation feature importance, we can see the green features that contribute the most to the drop of the overall model scores if the the feature's values are randomly shuffled. Those top features that are consistent in the darkest of green shades (in the normal & oversampled data) are only CLAGE & DELINQ feature. Special notice for NINQ & CLNO that came amongst the top in the oversampled data. In other words, all of these features are important for our model in predicting BAD categories.
4.5 [RANDOM FOREST REPORTS]¶
In [49]:
# model fitting for normal data
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(random_state= 190, n_estimators=1000)
rfc.fit(x_train, y_train)
Out[49]:
In [50]:
# model fitting for oversampled data
from sklearn.ensemble import RandomForestClassifier
rfc2 = RandomForestClassifier(random_state= 190, n_estimators=1000)
rfc2.fit(x_trainres, y_trainres)
Out[50]:
In [51]:
# classification report for normal data
rfc_pred = rfc.predict(x_test)
rfc_predprob = rfc.predict_proba(x_test)
print(classification_report(y_test, rfc_pred))
# classification report for oversampled data
rfc_pred2 = rfc2.predict(x_test)
rfc_predprob2 = rfc2.predict_proba(x_test)
print(classification_report(y_test, rfc_pred2))
In [56]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, rfc_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, rfc_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [47]:
# ROC - AUC Score for normal data
rfc_pred_proba = rfc.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, rfc_pred_proba)
auc = metrics.roc_auc_score(y_test, rfc_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
rfc_pred_proba2 = rfc2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, rfc_pred_proba2)
auc = metrics.roc_auc_score(y_test, rfc_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [51]:
# Cross Validation score for normal data
rfcscores = cross_val_score(estimator=rfc,
X=x_train,
y=y_train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(rfcscores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(rfcscores)), rfcscores)
plt.axhline(y=np.mean(rfcscores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
rfcscores2 = cross_val_score(estimator=rfc2,
X=x_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(rfcscores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(rfcscores2)), rfcscores2)
plt.axhline(y=np.mean(rfcscores2), color='g') # Mean value of cross validation scores
plt.show()
In [48]:
# Feature Importance for normal data
rfc_coef1 = pd.Series(rfc.feature_importances_, x_train.columns).sort_values(ascending= False)
rfc_coef1.plot(kind = 'bar', title='Feature Importances - Normal Data')
plt.show()
# Feature Importance for oversampled data
rfc_coef2 = pd.Series(rfc2.feature_importances_, x_trainres.columns).sort_values(ascending= False)
rfc_coef2.plot(kind = 'bar', title='Feature Importances - Oversampled Data')
plt.show()
- From the feature importance, we can see which features that contribute to the most information gained. Those top features that are consistent in the top three (in the normal & oversampled data) are DELINQ & CLAGE feature. Special notice for NINQ & VALUE that appears to approximate the top features in both data. In other words, all of these features are important for our model in predicting BAD categories.
In [49]:
# permutation importance for normal data
rfcperm = PermutationImportance(rfc, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(rfcperm, feature_names = list(x_test.columns))
Out[49]:
In [50]:
# permutation importance for oversampled data
rfcperm2 = PermutationImportance(rfc2, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(rfcperm2, feature_names = list(x_test.columns))
Out[50]:
- From the permutation feature importance, we can see the green features that contribute the most to the drop of the overall model scores if the the feature's values are randomly shuffled. Those top features that are consistent in the darkest of green shades (in the normal & oversampled data) are only CLAGE & DELINQ feature. Special notice for NINQ that came amongst the top in the oversampled data. In other words, all of these features are important for our model in predicting BAD categories.
4.6 [XGBOOST REPORTS]¶
In [59]:
# model fitting for normal data
from xgboost import XGBClassifier
xgb = XGBClassifier(learning_rate = 0.01, n_estimators = 10000, max_depth = None, n_jobs = -1)
xgb.fit(x_train, y_train)
Out[59]:
In [60]:
# model fitting for oversampled data
xgb2 = XGBClassifier(learning_rate = 0.01, n_estimators = 10000, max_depth = None, n_jobs = -1)
xgb2.fit(x_trainres, y_trainres)
Out[60]:
In [61]:
# classification report for normal data
xgb_pred = xgb.predict(x_test)
xgb_predprob = xgb.predict_proba(x_test)
print(classification_report(y_test, xgb_pred))
# classification report for oversampled data
xgb_pred2 = xgb2.predict(x_test)
xgb_predprob2 = xgb2.predict_proba(x_test)
print(classification_report(y_test, xgb_pred2))
In [62]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, xgb_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, xgb_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [56]:
# ROC - AUC Score for normal data
xgb_pred_proba = xgb.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, xgb_pred_proba)
auc = metrics.roc_auc_score(y_test, xgb_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
xgb_pred_proba2 = xgb2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, xgb_pred_proba2)
auc = metrics.roc_auc_score(y_test, xgb_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [60]:
# Cross Validation score for normal data
xgbscores = cross_val_score(estimator=xgb,
X=x_train,
y=y_train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(xgbscores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(xgbscores)), xgbscores)
plt.axhline(y=np.mean(xgbscores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
xgbscores2 = cross_val_score(estimator=xgb2,
X=x_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(xgbscores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(xgbscores2)), xgbscores2)
plt.axhline(y=np.mean(xgbscores2), color='g') # Mean value of cross validation scores
plt.show()
In [57]:
# Feature Importance for normal data
xgbcoef1 = pd.Series(xgb.feature_importances_, x_train.columns).sort_values(ascending= False)
xgbcoef1.plot(kind = 'bar', title='Feature Importances - Normal Data')
plt.show()
# Feature Importance for oversampled data
xgbcoef2 = pd.Series(xgb2.feature_importances_, x_trainres.columns).sort_values(ascending= False)
xgbcoef2.plot(kind = 'bar', title='Feature Importances - Oversampled Data')
plt.show()
- From the feature importance, we can see which features that contribute to the most information gained. Those top features that are consistent (in the normal & oversampled data) is DELINQ feature. Special notice for NINQ that appears to approximate the top features in both data. In other words, all of these features are important for our model in predicting BAD categories.
In [58]:
# permutation importance for normal data
xgbperm = PermutationImportance(xgb, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(xgbperm, feature_names = list(x_test.columns))
Out[58]:
In [59]:
# permutation importance for oversampled data
xgbperm2 = PermutationImportance(xgb2, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(xgbperm2, feature_names = list(x_test.columns))
Out[59]:
- From the permutation feature importance, we can see the green features that contribute the most to the drop of the overall model scores if the the feature's values are randomly shuffled. Those top features that are consistent in the darkest of green shades (in the normal & oversampled data) are only CLAGE & DELINQ feature. Special notice for MORTDUE that came amongst the top in both data. In other words, all of these features are important for our model in predicting BAD categories.
4.7 [NAIVE BAYES - BERNOULLI REPORTS]¶
In [63]:
# model fitting for normal data
from sklearn.naive_bayes import BernoulliNB
nb = BernoulliNB()
nb.fit(x_train, y_train)
Out[63]:
In [64]:
# model fitting for oversampled data
nb2 = BernoulliNB()
nb2.fit(x_trainres, y_trainres)
Out[64]:
In [65]:
# classification report for normal data
nb_pred = nb.predict(x_test)
nb_predprob = nb.predict_proba(x_test)
print(classification_report(y_test, nb_pred))
# classification report for oversampled data
nb_pred2 = nb2.predict(x_test)
nb_predprob2 = nb2.predict_proba(x_test)
print(classification_report(y_test, nb_pred2))
In [66]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, nb_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, nb_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [65]:
# ROC - AUC Score for normal data
nb_pred_proba = nb.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, nb_pred_proba)
auc = metrics.roc_auc_score(y_test, nb_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
nb_pred_proba2 = nb2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, nb_pred_proba2)
auc = metrics.roc_auc_score(y_test, nb_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [68]:
# Cross Validation score for normal data
nbscores = cross_val_score(estimator=nb,
X=x_train,
y=y_train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(nbscores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(nbscores)), nbscores)
plt.axhline(y=np.mean(nbscores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
nbscores2 = cross_val_score(estimator=nb2,
X=x_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(nbscores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(nbscores2)), nbscores2)
plt.axhline(y=np.mean(nbscores2), color='g') # Mean value of cross validation scores
plt.show()
In [66]:
# permutation importance for normal data
nbperm = PermutationImportance(nb, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(nbperm, feature_names = list(x_test.columns))
Out[66]:
In [67]:
# permutation importance for oversampled data
nbperm2 = PermutationImportance(nb2, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(nbperm2, feature_names = list(x_test.columns))
Out[67]:
- From the permutation feature importance, we can see the green features that contribute the most to the drop of the overall model scores if the the feature's values are randomly shuffled. Those top features that are consistent in the darkest of green shades (in the normal & oversampled data) is only the DELINQ feature. Special notice for NINQ & JOB_Office that came amongst the top in both data. In other words, all of these features are important for our model in predicting BAD categories.
4.8 [KNN REPORTS]¶
In [67]:
# creating new dummy variable for scaling
mydummy2 = mydummy.copy()
mydummy2.columns
Out[67]:
In [68]:
# importing standard scaler
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(mydummy2.drop('BAD',axis=1))
Out[68]:
In [69]:
# scaling features
scaled_features = scaler.transform(mydummy2.drop('BAD',axis=1))
df_feat = pd.DataFrame(scaled_features,columns=mydummy2.columns[1:])
df_feat.head()
Out[69]:
In [70]:
# splitting data
X_Train, X_Test, Y_Train, Y_Test = train_test_split(scaled_features,mydummy2['BAD'],
test_size=0.20, random_state= 101)
In [71]:
# SMOTE resampling
Y_Train = Y_Train.astype('int')
smo = SMOTE(random_state=0, sampling_strategy='minority')
X_Trainres, Y_Trainres = smo.fit_resample(X_Train, Y_Train)
print(sorted(Counter(Y_Trainres).items()))
In [72]:
# model fitting for normal data
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_Train, Y_Train)
Out[72]:
In [73]:
# model fitting for undersampled data
from sklearn.neighbors import KNeighborsClassifier
knn2 = KNeighborsClassifier(n_neighbors=3)
knn2.fit(X_Trainres, Y_Trainres)
Out[73]:
In [74]:
# classification report for normal data
knn_pred = knn.predict(X_Test)
knn_predprob = knn.predict_proba(X_Test)
print(classification_report(Y_Test, knn_pred))
# classification report for oversampled data
knn_pred2 = knn2.predict(X_Test)
knn_predprob2 = knn2.predict_proba(X_Test)
print(classification_report(Y_Test, knn_pred2))
In [75]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, knn_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, knn_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [76]:
# identifying error rates
error_rate = []
for i in range(1,40):
knn = KNeighborsClassifier(n_neighbors=i)
knn.fit(X_Train,Y_Train)
pred_i = knn.predict(X_Test)
error_rate.append(np.mean(pred_i != Y_Test))
error_rate2 = []
for i in range(1,40):
knn2 = KNeighborsClassifier(n_neighbors=i)
knn2.fit(X_Trainres,Y_Trainres)
pred_i2 = knn2.predict(X_Test)
error_rate2.append(np.mean(pred_i2 != Y_Test))
In [77]:
plt.figure(figsize=(10,6))
plt.plot(range(1,40),error_rate,color='blue', linestyle='dashed', marker='o',
markerfacecolor='red', markersize=10)
plt.title('Error Rate vs. K Value in Normal Data')
plt.xlabel('K')
plt.ylabel('Error Rate')
plt.show()
plt.figure(figsize=(10,6))
plt.plot(range(1,40),error_rate2,color='blue', linestyle='dashed', marker='o',
markerfacecolor='red', markersize=10)
plt.title('Error Rate vs. K Value in Oversampled Data')
plt.xlabel('K')
plt.ylabel('Error Rate')
plt.show()
In [79]:
from sklearn import metrics
# ROC - AUC Score for normal data
knn_pred_proba = knn.predict_proba(X_Test)[::,1]
fpr, tpr, _ = metrics.roc_curve(Y_Test, knn_pred_proba)
auc = metrics.roc_auc_score(Y_Test, knn_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
knn_pred_proba2 = knn2.predict_proba(X_Test)[::,1]
fpr, tpr, _ = metrics.roc_curve(Y_Test, knn_pred_proba2)
auc = metrics.roc_auc_score(Y_Test, knn_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [83]:
# Cross Validation score for normal data
knnscores = cross_val_score(estimator=knn,
X=X_Train,
y=Y_Train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(knnscores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(knnscores)), knnscores)
plt.axhline(y=np.mean(knnscores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
knnscores2 = cross_val_score(estimator=knn2,
X=X_Trainres,
y=Y_Trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(knnscores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(knnscores2)), knnscores2)
plt.axhline(y=np.mean(knnscores2), color='g') # Mean value of cross validation scores
plt.show()
In [81]:
# permutation importance for normal data
knnperm = PermutationImportance(knn, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(knnperm, feature_names = list(x_test.columns))
Out[81]:
In [82]:
# permutation importance for oversampled data
knnperm2 = PermutationImportance(knn2, scoring = 'roc_auc', random_state= 101).fit(x_test, y_test)
show_weights(knnperm2, feature_names = list(x_test.columns))
Out[82]:
- From the permutation feature importance, we can see the green features that contribute the most to the drop of the overall model scores if the the feature's values are randomly shuffled. Those top features that are consistent in the darkest of green shades (in the normal & oversampled data) is only the MORTDUE feature. Special notice for VALUE & CLAGE that came amongst the top in the oversampled data. In other words, all of these features are important for our model in predicting BAD categories.
4.9 [LOGISTIC REGRESSION REPORTS]¶
In [80]:
from sklearn.linear_model import LogisticRegression
# fit the model with normal data
logreg = LogisticRegression()
logreg.fit(x_train, y_train)
Out[80]:
In [81]:
# fit the model with oversampled data
logreg2 = LogisticRegression()
logreg2.fit(x_trainres, y_trainres)
Out[81]:
In [82]:
# classification report for normal data
from sklearn.metrics import classification_report
logreg_pred = logreg.predict(x_test)
logreg_predprob = logreg.predict_proba(x_test)
print(classification_report(y_test, logreg_pred))
# classification report for oversampled data
logreg_pred2 = logreg2.predict(x_test)
logreg_predprob2 = logreg2.predict_proba(x_test)
print(classification_report(y_test, logreg_pred2))
In [83]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, logreg_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, logreg_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [84]:
from sklearn import metrics
print('Matthews Correlation Coefficient - Normal Data: {}'.format(metrics.matthews_corrcoef(y_test, logreg_pred)))
print('Matthews Correlation Coefficient - Oversampled Data: {}'.format(metrics.matthews_corrcoef(y_test, logreg_pred2)))
print('Log Loss - Normal Data: {}'.format(metrics.log_loss(y_test, logreg_predprob)))
print('Log Loss - Oversampled Data: {}'.format(metrics.log_loss(y_test, logreg_predprob2)))
In [92]:
# ROC - AUC Score for normal data
logreg_pred_proba = logreg.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, logreg_pred_proba)
auc = metrics.roc_auc_score(y_test, logreg_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled data
logreg_pred_proba2 = logreg2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, logreg_pred_proba2)
auc = metrics.roc_auc_score(y_test, logreg_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [85]:
import statsmodels.api as sm
# model summary for normal data
logit_model = sm.Logit(y_train, sm.add_constant(x_train))
result = logit_model.fit(method = 'lbfgs')
print('##### Model Summary - Normal Data: #####')
print('\n')
print(result.summary2())
print('\n')
# model summary for oversampled data
logit_model2 = sm.Logit(y_trainres, sm.add_constant(x_trainres))
result2 = logit_model.fit(method = 'lbfgs')
print('##### Model Summary - Oversampled Data: #####')
print('\n')
print(result2.summary2())
- It seems that our logreg model is not yet able to read our data well, it is still random guessing. Perhaps, we need to have some data transformation
In [86]:
from sklearn.preprocessing import FunctionTransformer
transformer = FunctionTransformer(np.log1p, validate = True)
X_train = x_train.copy()
X_trainres = x_trainres.copy()
# transforming features in normal data with log1p
for i in x_train.columns:
X_train[i] = transformer.fit_transform(np.array(X_train[i]).reshape(1,-1))[0]
# transforming features in oversampled data with log1p
for j in x_trainres.columns:
X_trainres[j] = transformer.fit_transform(np.array(X_trainres[j]).reshape(1,-1))[0]
In [87]:
X_train.head()
Out[87]:
In [88]:
# fit the model with normal transformed data
logregt = LogisticRegression()
logregt.fit(X_train, y_train)
Out[88]:
In [89]:
# fit the model with oversampled data
logregt2 = LogisticRegression()
logregt2.fit(X_trainres, y_trainres)
Out[89]:
In [90]:
# classification report for normal transformed data
logregt_pred = logregt.predict(x_test)
logregt_predprob = logregt.predict_proba(x_test)
print(classification_report(y_test, logregt_pred))
# classification report for oversampled transformed data
logregt_pred2 = logregt2.predict(x_test)
logregt_predprob2 = logregt2.predict_proba(x_test)
print(classification_report(y_test, logregt_pred2))
In [91]:
# confusion matrix for normal transformed data
cnf_matrix = confusion_matrix(y_test, logregt_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled transformed data
cnf_matrix = confusion_matrix(y_test, logregt_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [92]:
print('Matthews Correlation Coefficient - Normal Transformed Data: {}'.format(metrics.matthews_corrcoef(y_test, logregt_pred)))
print('Matthews Correlation Coefficient - Oversampled Transformed Data: {}'.format(metrics.matthews_corrcoef(y_test, logregt_pred2)))
print('Log Loss - Normal Transformed Data: {}'.format(metrics.log_loss(y_test, logregt_predprob)))
print('Log Loss - Oversampled Transformed Data: {}'.format(metrics.log_loss(y_test, logregt_predprob2)))
In [100]:
# ROC - AUC Score for normal transformed data
logregt_pred_proba = logregt.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, logregt_pred_proba)
auc = metrics.roc_auc_score(y_test, logregt_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Normal & Transformed')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for oversampled transformed data
logregt_pred_proba2 = logregt2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, logregt_pred_proba2)
auc = metrics.roc_auc_score(y_test, logregt_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - Oversampled & Transformed')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [96]:
from sklearn.model_selection import cross_val_score
# Cross Validation score for normal data
logregtscores = cross_val_score(estimator=logregt,
X=X_train,
y=y_train,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross validation - Normal Data scores: {}'.format(logregtscores))
plt.title('Cross Validation - Normal Data')
plt.scatter(np.arange(len(logregtscores)), logregtscores)
plt.axhline(y=np.mean(logregtscores), color='g') # Mean value of cross validation scores
plt.show()
# Cross Validation score for oversampled data
logregtscores2 = cross_val_score(estimator=logregt2,
X=X_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(logregtscores2))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(logregtscores2)), logregtscores2)
plt.axhline(y=np.mean(logregtscores2), color='g') # Mean value of cross validation scores
plt.show()
In [93]:
# model summary for normal transformed data
logit_model_t = sm.Logit(y_train, sm.add_constant(X_train))
result_t = logit_model_t.fit(method = 'lbfgs')
print('##### Model Summary - Normal Data: #####')
print('\n')
print(result_t.summary2())
print('\n')
# model summary for oversampled transformed data
logit_mode_t2 = sm.Logit(y_trainres, sm.add_constant(X_trainres))
result_t2 = logit_mode_t2.fit(method = 'lbfgs')
print('##### Model Summary - Oversampled Data: #####')
print('\n')
print(result_t2.summary2())
- From the model summary, we can see the features that has influence on our target variable (BAD) by looking at their p-values. If the p-value < 0.05, it means that the feature have influence over BAD. For better interpretability & relevance, I choose to interpret only the normal & transformed data. And from the observations, we can see that the features that has influence over the BAD categories are:
- DELINQ - CLAGE - NINQ - LOAN - VALUE - JOB_Office - REASON_HomeImp - It is interesting to note that DELINQ feature has the biggest coefficient value amongst the features with the value of 1.6046. This finding could show that in overall DELINQ has a bigger influence towards BAD in relative to other features.
5. [OVERALL MODEL PERFORMANCE]¶
In [103]:
# Finding the highest accuracy & recall score (trying to minimize False Negative)
list1 = [dtree_pred, dtree_pred2, rfc_pred, rfc_pred2, xgb_pred, xgb_pred2, knn_pred, knn_pred2, nb_pred, nb_pred2, logreg_pred, logreg_pred2,
logregt_pred, logregt_pred2,]
list3 = ['DT :', 'DT2 :', 'RF :', 'RF2 :', 'XGB :', 'XGB2:', 'KNN :', 'KNN2:', 'NB :', 'NB2 :',
'LR :', 'LR2 :', 'LRT :', 'LRT2:' ]
for i,j in list(zip(list1,list3)):
print('Accuracy Score in',j,metrics.accuracy_score(y_test, i))
print(' Recall Score in',j,metrics.recall_score(y_test, i))
The highest overall Accuracy & Recall score is achieved by KNN2 (K-NearestNeighboursClassifier - Oversampled) with an Accuracy Score of 0.9647727272727272 & Recall Score of 0.8440860215053764.
- It means that the model predicted 97% of data rightly & recalled 84% of all the Actual Positives
The second highest overall Accuracy & Recall score is achieved by RF2 (RandomForestClassifier - Oversampled) with an Accuracy Score of 0.9409090909090909 & Recall Score of 0.7473118279569892.
- It means that the model predicted 94% of data rightly & recalled 75% of all the Actual Positives In [104]:
# Finding the highest ROC-AUC score
list2 = [dtree_pred_proba, dtree_pred_proba2, rfc_pred_proba, rfc_pred_proba2, xgb_pred_proba, xgb_pred_proba2, knn_pred_proba, knn_pred_proba2, nb_pred_proba, nb_pred_proba2, logreg_pred_proba, logreg_pred_proba2,
logregt_pred_proba, logregt_pred_proba2,]
list3 = ['DT :', 'DT2 :', 'RF :', 'RF2 :', 'XGB :', 'XGB2:', 'KNN :', 'KNN2:', 'NB :', 'NB2 :',
'LR :', 'LR2 :', 'LRT :', 'LRT2:' ]
for i,j in list(zip(list2,list3)):
print('ROC-AUC Score in',j,metrics.roc_auc_score(y_test, i))
- The highest ROC-AUC score is achieved by RF (RandomForestClassifier - Normal) with an ROC-AUC score of 0.9901808124941899
- The second highest ROC-AUC score is achieved by RF2 (RandomForestClassifier - Oversampled) with an ROC-AUC score of 0.9875546155991448
In [105]:
# Finding the highest average of Cross-Validation scores (cv=10)
list1 = [dtreescores, dtreescores2, rfcscores, rfcscores2, xgbscores, xgbscores2, knnscores, knnscores2, nbscores, nbscores2, logregtscores, logregtscores2]
list3 = ['DT :', 'DT2 :', 'RF :', 'RF2 :', 'XGB :', 'XGB2:', 'KNN :', 'KNN2:', 'NB :', 'NB2 :',
'LRT :', 'LRT2:']
for i,j in list(zip(list1,list3)):
print('Average Cross-Validation score in',j,sum(i)/len(i))
- The highest Average Cross-Validation score is achieved by RF2 (RandomForestClassifier - Oversampled) with an average score of 0.9958538683625295
- The second highest Average Cross-Validation score is achieved by XGB2 (XGBoostingClassifier - Oversampled) with an average score of 0.9920089258233302
6. [HYPERPARAMETER TUNING]¶
- From the previous model evaluations, we may see that RandomForestClassifier (Oversampled) is, in overall, the most consistent in predicting our home equity loan default dataset. From here on, we will conduct hyperparameter tuning using RandomForestClassifier (Oversampled) model.
In [106]:
## RANDOMIZEDSEARCHCV
from sklearn.model_selection import RandomizedSearchCV
# number of tress in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# number of features to consider at every split
max_features = ['auto','sqrt']
# maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# minimum number of samples required to split a node
min_samples_split = [int(x) for x in np.linspace(start = 100, stop = 1000, num = 100)]
# minimum number of samples required at each leaf node
min_samples_leaf = [int(x) for x in np.linspace(1, 11, num = 10)]
# method of selecting samples for training each tree
bootstrap = [True, False]
# create the random grid
random_grid = {'n_estimators': n_estimators,
'max_features': max_features,
'max_depth': max_depth,
'min_samples_split': min_samples_split,
'min_samples_leaf': min_samples_leaf,
'bootstrap': bootstrap}
In [107]:
# making three sets of RandomizedSearch
rf_random = RandomizedSearchCV(estimator = rfc,
param_distributions = random_grid,
cv = 3, n_jobs = -1, n_iter = 10)
rf_random2 = RandomizedSearchCV(estimator = rfc,
param_distributions = random_grid,
cv = 3, n_jobs = -1, n_iter = 10)
rf_random3 = RandomizedSearchCV(estimator = rfc,
param_distributions = random_grid,
cv = 3, n_jobs = -1, n_iter = 10)
In [108]:
# first set
rf_random.fit(x_trainres, y_trainres)
Out[108]:
In [109]:
# first set's best params
rf_random.best_params_
Out[109]:
In [110]:
# second set
rf_random2.fit(x_trainres, y_trainres)
Out[110]:
In [111]:
# second set's best param
rf_random2.best_params_
Out[111]:
In [112]:
# third set
rf_random3.fit(x_trainres, y_trainres)
Out[112]:
In [113]:
# third set's best param
rf_random3.best_params_
Out[113]:
- rfc2 parameters:
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
criterion='gini', max_depth=None, max_features='auto', max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=1000, n_jobs=None, oob_score=False, random_state=190, verbose=0, warm_start=False)
In [114]:
## GRIDSEARCHCV
from sklearn.model_selection import GridSearchCV
# using estimator of rfc2, our best & most consistent model
# the hyperparameter input here is fitted in accordance with best hyperparameters that we have found in the previous model as well as its randomized searches
# since we are focusing on its ability to avoid False Negatives, we use recall as its gridsearchcv scoring
grid = GridSearchCV(estimator = rfc2,
refit = 'recall',
param_grid = {
'n_estimators':[1000,1800],
'bootstrap': [True,False],
'max_features': ['sqrt','auto'],
'max_depth': [80,None],
},
scoring = 'recall',
cv = 5, n_jobs = -1)
In [115]:
grid.fit(x_trainres, y_trainres)
Out[115]:
In [116]:
# best score
grid.best_score_
Out[116]:
In [117]:
# best parameters
grid.best_params_
Out[117]:
In [7]:
# creating & fitting a new RFC model with the best parameters from GridSearch
from sklearn.ensemble import RandomForestClassifier
tuned = RandomForestClassifier(bootstrap= False, max_depth = 80, max_features = 'sqrt', n_estimators = 1800, random_state = 1)
tuned.fit(x_trainres, y_trainres)
Out[7]:
In [99]:
print("-------------BEFORE HYPERPARAMETER TUNING-------------")
print(classification_report(y_test,rfc_pred2))
tuned_pred = tuned.predict(x_test)
tuned_predprob = tuned.predict_proba(x_test)
print("\n")
print("--------------AFTER HYPERPARAMETER TUNING-------------")
print(classification_report(y_test,tuned_pred))
In [120]:
print("-------------BEFORE HYPERPARAMETER TUNING-------------")
print('Accuracy Score',metrics.accuracy_score(y_test, rfc_pred2))
print(' Recall Score',metrics.recall_score(y_test, rfc_pred2))
print(' F1 Score',metrics.f1_score(y_test, rfc_pred2))
print("\n")
print("-------------AFTER HYPERPARAMETER TUNING---------------")
print('Accuracy Score',metrics.accuracy_score(y_test, tuned_pred))
print(' Recall Score',metrics.recall_score(y_test, tuned_pred))
print(' F1 Score',metrics.f1_score(y_test, tuned_pred))
In [100]:
# confusion matrix for after hyperparameter tuning
cnf_matrix = confusion_matrix(y_test, tuned_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [135]:
# ROC - AUC Score for after hyperparameter tuning
tuned_pred_proba = tuned.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, tuned_pred_proba)
auc = metrics.roc_auc_score(y_test, tuned_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - After Hyperparameter Tuning')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [122]:
# Cross Validation score for tuned & oversampled data
tunedscores = cross_val_score(estimator=tuned,
X=x_trainres,
y=y_trainres,
cv=10,
n_jobs=1,
scoring = 'roc_auc')
print('Cross Validation - Oversampled Data scores: {}'.format(tunedscores))
plt.title('Cross Validation - Oversampled Data')
plt.scatter(np.arange(len(tunedscores)), tunedscores)
plt.axhline(y=np.mean(tunedscores), color='g') # Mean value of cross validation scores
plt.show()
print('Average value of cross validation scores: ',np.mean(tunedscores))
- it can be seen that our overall model has improved, especially its recall scoring (for the positive value) from 0.75 to 0.80
- with an increase in recall scoring, the F1 score also improved from 0.84 to 0.88 and the overall accuracy increased 1% to 0.95
- it also increase the ROC-AUC score from 0.9901 to 0.9923
- hyperparameter tuning does increase the overall performance of the RandomForestClassifier (Oversampled) model
7. [LEARNING CURVE]¶
In [140]:
from sklearn.model_selection import learning_curve
train_sizes, train_scores, test_scores = learning_curve(estimator=tuned,
X=x,
y=y,
train_sizes=np.linspace(0.5, 1.0, 5),
cv=10)
# Mean value of accuracy against training data
train_mean = np.mean(train_scores, axis=1)
print('train mean: ')
print(train_mean)
# Standard deviation of training accuracy per number of training samples
train_std = np.std(train_scores, axis=1)
# Same as above for test data
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)
print('test mean: ')
print(test_mean)
# Plot training accuracies
plt.plot(train_sizes, train_mean, color='red', marker='o', label='Training Accuracy')
# Plot the variance of training accuracies
plt.fill_between(train_sizes,
train_mean + train_std,
train_mean - train_std,
alpha=0.15, color='red')
# Plot for test data as training data
plt.plot(train_sizes, test_mean, color='blue', linestyle='--', marker='s',
label='Test Accuracy')
plt.fill_between(train_sizes,
test_mean + test_std,
test_mean - test_std,
alpha=0.15, color='blue')
plt.xlabel('Number of training samples')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
- From the learning curve, we may see that our tuned model gains more test accuracy as we increase the number of training samples (with the condition that our training accuracy is stable). This curve means that our tuned model may learn & perform better with bigger dataset. Perhaps, we could give it more datasets in the future so that it could gain better accuracy.
8. [AUTO ML REPORTS]¶
In [12]:
# let's benchmark our tuned model performance with AutoML
from tpot import TPOTClassifier
tpot = TPOTClassifier(subsample = 0.8, verbosity = 2, warm_start=True, early_stop=20, max_time_mins= 60, n_jobs= -2)
In [41]:
# fitting TPOT to our data
tpot.fit(x_train, y_train)
Out[41]:
In [62]:
# exporting our model results
tpot.export('tpot_LOAN_DEFAULT.py')
In [63]:
# using our AutoML model ('tpot_LOAN_DEFAULT.py') to normal & oversampled data
import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import BernoulliNB
from sklearn.pipeline import make_pipeline, make_union
from sklearn.preprocessing import MinMaxScaler, PolynomialFeatures
from sklearn.tree import DecisionTreeClassifier
from tpot.builtins import StackingEstimator
# Average CV score on the training set was: 0.9353463587921848
# fitting the AutoML for the normal data
exported_pipeline = make_pipeline(
PolynomialFeatures(degree=2, include_bias=False, interaction_only=False),
StackingEstimator(estimator=ExtraTreesClassifier(bootstrap=False, criterion="gini", max_features=0.55, min_samples_leaf=2, min_samples_split=5, n_estimators=100)),
MinMaxScaler(),
StackingEstimator(estimator=BernoulliNB(alpha=0.001, fit_prior=True)),
DecisionTreeClassifier(criterion="entropy", max_depth=5, min_samples_leaf=12, min_samples_split=9))
exported_pipeline.fit(x_train, y_train)
automl = exported_pipeline.predict(x_test)
# fitting the AutoML for the oversampled data
exported_pipeline2 = make_pipeline(
PolynomialFeatures(degree=2, include_bias=False, interaction_only=False),
StackingEstimator(estimator=ExtraTreesClassifier(bootstrap=False, criterion="gini", max_features=0.55, min_samples_leaf=2, min_samples_split=5, n_estimators=100)),
MinMaxScaler(),
StackingEstimator(estimator=BernoulliNB(alpha=0.001, fit_prior=True)),
DecisionTreeClassifier(criterion="entropy", max_depth=5, min_samples_leaf=12, min_samples_split=9))
exported_pipeline2.fit(x_trainres, y_trainres)
automl2 = exported_pipeline2.predict(x_test)
In [64]:
# classification report for autoML & normal data
from sklearn.metrics import classification_report,confusion_matrix
print(classification_report(y_test, automl))
# classification report for autoML & oversampled data
print(classification_report(y_test, automl2))
In [65]:
from sklearn import metrics
# ROC - AUC Score for autoML in normal data
automl_pred_proba = exported_pipeline.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, automl_pred_proba)
auc = metrics.roc_auc_score(y_test, automl_pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - AutoML & Normal Data')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
# ROC - AUC Score for autoML in oversampled data
automl_pred_proba2 = exported_pipeline2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, automl_pred_proba2)
auc = metrics.roc_auc_score(y_test, automl_pred_proba2)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - AutoML & Oversampled Data')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
From the observations:
- If we compare the results that we get from autoML & our own tuned model (oversampled), we could see that the latter performs better. Our own model performs higher not only in accuracy & recall scores but also in ROC-AUC Scores.
- It is better for the analyzing counterpart to use the tuned model (oversampled) for better overall prediction
9. [MODEL DEPLOYMENT]¶
In [17]:
# fit our tuned model to the whole of original dataframe
tuned.fit(x,y)
Out[17]:
In [20]:
# prediction results
from sklearn.metrics import classification_report,confusion_matrix
all_pred = tuned.predict(x)
all_predprob = tuned.predict_proba(x)
print(classification_report(y, all_pred))
In [21]:
# saving algorithm for further usage
import pickle
filename = 'hmeq_loan_default_tuned.sav';
pickle.dump(tuned, open(filename, 'wb'))
10. [CONCLUDING NOTES]¶
1) On features:¶
Through all the tests & observations, there are three most frequent features that keeps showing up and they are DELINQ, CLAGE, and NINQ features with DELINQ as the most fundamental feature. These feature(s) could be the feature(s) that has better association towards our target variable (BAD) and may prove to be important for further tests & observations.
2) On clusters:¶
There's a cluster of outliers that is consisted of 37 datapoints & has an alarming sign towards loan default. In relative to other customers, the bank needs further precaution for this type of customer.
3) On predictions:¶
In overall, the best model for our prediction is RandomForestClassifier (Oversampled) that's hyperparameter-tuned. It even gives better results than the AutoML model, especially in its Recall since we're trying to minimize False Negatives in our predictions.
4) On samples:¶
There's an indication that the model predictions could improve with larger datasets. It's recommended for further tests & observations to have bigger datasets.